In [1]:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
In [2]:
!wget http://www.cs.colostate.edu/~anderson/cs480/notebooks/nn2.tar
!tar xvf nn2.tar
In [2]:
import neuralnetworks as nn
import qdalda
import mlutils as ml
Now let's go through the steps of applying a neural network classifier to the Student Alcohol Consumption data set, recently added to the UCI ML Repository. This data set has 1,044 samples, each with 33 attributes. The paper Using data Mining to Predict Secondary School Student Alcohol Consumption describes the application of random forests to this classification problem. This paper will serve as our guide on how to set up the data.
Start by downloading the student.zip file, either by clicking on the link in the Data Folder at the Repository, or by using wget.
In [3]:
!wget http://archive.ics.uci.edu/ml/machine-learning-databases/00356/student.zip
!unzip -o student.zip
!rm student.zip
!ls student*
In [4]:
!cat student.txt
In [6]:
!head -10 student-mat.csv
So, first of all, we see that semi-colons are the field separator, and the first line is column headings. Also, notice that the non-numeric values are surrounded by double quotes. And the third and second to last numeric fields are also surrounded by double quotes.
Read the first line into a list of column names.
In [7]:
with open('student-mat.csv') as file:
headings = file.readline().split(';')
headings = [head.strip() for head in headings]
print(headings)
Now, how do we read the data lines? My usual answer is to use np.loadtxt. The Pandas package is something you should read about and practice using in the future.
np.loadtxt allows you to specify a function to call to convert values in each column. The converters are specified as a dictionary with keys for column indices. Here is a definition of converters for each column interspersed with the data set's documentation for each attribute.
In [8]:
# 1 school - student's school (binary: 'GP' - Gabriel Pereira or 'MS' - Mousinho da Silveira)
school = {'GP':0, 'MS':1}
# 2 sex - student's sex (binary: 'F' - female or 'M' - male)
gender = {'F':0, 'M':1}
# 3 age - student's age (numeric: from 15 to 22)
# 4 address - student's home address type (binary: 'U' - urban or 'R' - rural)
address = {'U':0, 'R':1}
# 5 famsize - family size (binary: 'LE3' - less or equal to 3 or 'GT3' - greater than 3)
famsize = {'LE3':0, 'GT3':1}
# 6 Pstatus - parent's cohabitation status (binary: 'T' - living together or 'A' - apart)
parentsStatus = {'T':0, 'A':1}
# 7 Medu - mother's education (numeric: 0 - none, 1 - primary education (4th grade), 2 – 5th to 9th grade, 3 – secondary education or 4 – higher education)
# 8 Fedu - father's education (numeric: 0 - none, 1 - primary education (4th grade), 2 – 5th to 9th grade, 3 – secondary education or 4 – higher education)
# 9 Mjob - mother's job (nominal: 'teacher', 'health' care related, civil 'services' (e.g. administrative or police), 'at_home' or 'other')
parentJob = {'teacher':0, 'health':1, 'services':2, 'at_home':3, 'other':4}
# 10 Fjob - father's job (nominal: 'teacher', 'health' care related, civil 'services' (e.g. administrative or police), 'at_home' or 'other')
# 11 reason - reason to choose this school (nominal: close to 'home', school 'reputation', 'course' preference or 'other')
reason = {'home':0, 'reputation':1, 'course':2, 'other':3}
# 12 guardian - student's guardian (nominal: 'mother', 'father' or 'other')
guardian = {'mother':0, 'father':1, 'other':2}
# 13 traveltime - home to school travel time (numeric: 1 - <15 min., 2 - 15 to 30 min., 3 - 30 min. to 1 hour, or 4 - >1 hour)
# 14 studytime - weekly study time (numeric: 1 - <2 hours, 2 - 2 to 5 hours, 3 - 5 to 10 hours, or 4 - >10 hours)
# 15 failures - number of past class failures (numeric: n if 1<=n<3, else 4)
# 16 schoolsup - extra educational support (binary: yes or no)
noYes = {'no':0, 'yes':1}
# 17 famsup - family educational support (binary: yes or no)
# 18 paid - extra paid classes within the course subject (Math or Portuguese) (binary: yes or no)
# 19 activities - extra-curricular activities (binary: yes or no)
# 20 nursery - attended nursery school (binary: yes or no)
# 21 higher - wants to take higher education (binary: yes or no)
# 22 internet - Internet access at home (binary: yes or no)
# 23 romantic - with a romantic relationship (binary: yes or no)
# 24 famrel - quality of family relationships (numeric: from 1 - very bad to 5 - excellent)
# 25 freetime - free time after school (numeric: from 1 - very low to 5 - very high)
# 26 goout - going out with friends (numeric: from 1 - very low to 5 - very high)
# 27 Dalc - workday alcohol consumption (numeric: from 1 - very low to 5 - very high)
# 28 Walc - weekend alcohol consumption (numeric: from 1 - very low to 5 - very high)
# 29 health - current health status (numeric: from 1 - very bad to 5 - very good)
# 30 absences - number of school absences (numeric: from 0 to 93)
# # these grades are related with the course subject, Math or Portuguese:
# 31 G1 - first period grade (numeric: from 0 to 20)
# 31 G2 - second period grade (numeric: from 0 to 20)
# 32 G3 - final grade (numeric: from 0 to 20, output target)
def bs(bytes):
'''Convert bytes to string and remove double quotes'''
# print(bytes)
return str(bytes,'utf-8').replace('"','')
converters = {0: lambda x: school[bs(x)],
1: lambda x: gender[bs(x)],
2: lambda age: age,
3: lambda x: address[bs(x)],
4: lambda x: famsize[bs(x)],
5: lambda x: parentsStatus[bs(x)],
6: lambda motherEd: motherEd,
7: lambda fatherEd: fatherEd,
8: lambda m: parentJob[bs(m)],
9: lambda f: parentJob[bs(f)],
10: lambda x: reason[bs(x)],
11: lambda x: guardian[bs(x)],
12: lambda travelTime: travelTime,
13: lambda studyTime: studyTime,
14: lambda failures: failures,
15: lambda schoolSupport: noYes[bs(schoolSupport)],
16: lambda famsup: noYes[bs(famsup)],
17: lambda paid: noYes[bs(paid)],
18: lambda activities: noYes[bs(activities)],
19: lambda nursery: noYes[bs(nursery)],
20: lambda higher: noYes[bs(higher)],
21: lambda internet: noYes[bs(internet)],
22: lambda romantic: noYes[bs(romantic)],
23: lambda famrel: famrel,
24: lambda freetime: freetime,
25: lambda goout: goout,
26: lambda workdayAlcohol: workdayAlcohol,
27: lambda weekendAlcohol: weekendAlcohol,
28: lambda health: health,
29: lambda absences: absences,
30: lambda G1: bs(G1),
31: lambda G2: bs(G2),
32: lambda G3: G3}
In [9]:
converters[11](b'mother'), converters[11](b'father')
Out[9]:
Now we can read in both files.
In [10]:
with open('student-mat.csv') as file:
headings = file.readline().split(';')
headings = [head.strip() for head in headings]
datamat = np.loadtxt(file, delimiter=';',converters=converters)
datapor = np.loadtxt('student-por.csv', delimiter=';',skiprows=1,converters=converters)
In [11]:
datamat.shape, datapor.shape
Out[11]:
These two data sets are for two different tests, a math test and a portugese test. Let's combine them, and add a first column with value 0 for math and 1 for portugese.
In [12]:
# add first column to indicate which subject
headings = ['subject'] + headings
datamat = np.hstack((np.zeros((datamat.shape[0],1)), datamat))
datapor = np.hstack((np.ones((datapor.shape[0],1)), datapor))
data = np.vstack((datamat,datapor))
data.shape
Out[12]:
Now we can pull out the two columns that indicate alcohol use. Which ones are they?
In [13]:
[i for i,head in enumerate(headings) if 'alc' in head]
Out[13]:
In [14]:
headings[27:29]
Out[14]:
In [15]:
data[:5,27:29]
Out[15]:
Let's look at these values. Add a bit of random noise to each to make them visible.
In [16]:
alcoholColumns = [26, 27]
T2 = data[:,alcoholColumns]
T2r = T2 + np.random.normal(0,0.1,size=T2.shape)
plt.plot(T2r[:,0],T2r[:,1], 'o', alpha=0.5);
For the experiments described in the paper linked above these two values are combined into one, then the value is discretized into two binary values, 0 for low alcohol use and 1 for high alcohol use. A weighted average of the two is formed, with weekend consumption (the second value) more heavily weighted. Then this value is quantized to 0 if the weighted average is less than 2, 1 otherwise. In the paper, a cutoff of 3 was used.
In [17]:
wavg = (T2[:,0:1] * 2 + T2[:,1:2] * 5) / 7
T = (wavg > 2.0).astype(int)
plt.subplot(1,2,1)
plt.hist(wavg)
plt.subplot(1,2,2)
plt.hist(T);
Now, for the input $X$ matrix.
In [18]:
X = data.copy()
X = np.delete(X, alcoholColumns, axis=1)
Xnames = [h for h in headings if 'alc' not in h]
print(len(Xnames),Xnames)
X.shape, T.shape
Out[18]:
Now we are ready to do some classifying. Let's use a small neural network.
In [19]:
nnet = nn.NeuralNetworkClassifier(X.shape[1], [2], len(np.unique(T)))
nnet.train(X,T,nIterations=100,verbose=True)
predictedClass = nnet.use(X)
plt.plot(np.exp(-nnet.getErrorTrace()))
accuracy = np.sum(predictedClass == T)/ len(T) * 100
print('Accuracy',accuracy)
Let's try different sized nets.
In [20]:
for hidden in [0,1,2,5,10]:
nnet = nn.NeuralNetworkClassifier(X.shape[1], [hidden], len(np.unique(T)))
nnet.train(X,T,nIterations=100,verbose=False)
predictedClass = nnet.use(X)
plt.plot(np.exp(-nnet.getErrorTrace()), label='h='+str(hidden))
plt.legend(loc='best')
accuracy = np.sum(predictedClass == T)/ len(T) * 100
print(hidden,accuracy)
What about deeper nets?
In [21]:
plt.figure(figsize=(15,6))
for layers in [1,2,3]:
plt.subplot(1,3,layers)
for hidden in [1,2,5,10]:
hiddens = [hidden]*layers
nnet = nn.NeuralNetworkClassifier(X.shape[1], hiddens, len(np.unique(T)))
nnet.train(X,T,nIterations=100,verbose=False)
predictedClass = nnet.use(X)
plt.plot(np.exp(-nnet.getErrorTrace()), label='h='+str(hidden))
plt.legend(loc='best')
accuracy = np.sum(predictedClass == T)/ len(T) * 100
print(hiddens,accuracy)
plt.title('{} layers'.format(layers))
What patterns are the neural networks finding in the data that allow them to classify this data?
One way to answer this is to look at the first hidden layer weight vectors. We can see this in the network diagram.
In [32]:
nnet = nn.NeuralNetworkClassifier(X.shape[1],[1],len(np.unique(T)))
nnet.train(X,T,nIterations = 500)
plt.plot(np.exp(-nnet.getErrorTrace()))
Out[32]:
In [33]:
plt.figure(figsize=(6,8))
nnet.draw(inputNames = Xnames)
The largest magnitude weight in the hidden layer is the positive weight on goout on the single hidden unit. goout has values of 1 to 5, with higher values representing more frequently going out. The hidden unit is connected negatively to the output. Considering these weights in isolation, the classifier output is decreased when the goout is higher than average. When goout is lower than average, its standardized value is negative. When multiplied by its positive weight, its contribution to the sum in the second unit is negative. This decreases the value of the output of the unit, when increases the output value sum.
So, staying home is correlated with more alcohol use. Make sense?
No, it doesn't. What's wrong with our reasoning?
This network has one output unit, but remember, there are two classes. The probability of the second class for a sample is 1 minus the probability of this single output unit. So, the output unit in this network outputs estimates the probability of class 0, which represents low alcohol use.
So, now it all makes sense. goout is positively connected to the hidden unit, which is negatively connected to the low alcohol class output, or goout is negatively correlated to low alcohol use.
Let's check this by plotting the output class, 0 or 1, versus goout.
In [41]:
plt.plot(X[:,-6]+np.random.normal(0,0.05,size=X[:,-6].shape), T+np.random.normal(0,0.05,size=T.shape),'o',alpha=0.2)
plt.xlabel('goout')
plt.ylabel('alcohol');
Yep, it does look like alcohol use is positively correlated with gooout.
In [42]:
hidden = 2
plt.figure(figsize=(10,8))
nnet = nn.NeuralNetworkClassifier(X.shape[1],[hidden],len(np.unique(T)))
nnet.train(X,T,nIterations = 1000)
plt.subplot(1,2,1)
plt.plot(np.exp(-nnet.getErrorTrace()))
plt.subplot(1,2,2)
nnet.draw(inputNames = Xnames)
In [43]:
hidden = 5
plt.figure(figsize=(10,8))
nnet = nn.NeuralNetworkClassifier(X.shape[1],[hidden],len(np.unique(T)))
nnet.train(X,T,nIterations = 1000)
plt.subplot(1,2,1)
plt.plot(np.exp(-nnet.getErrorTrace()))
plt.subplot(1,2,2)
nnet.draw(inputNames = Xnames)
We could also take a look at extreme samples, such as the sample that produces the highest probability of high alcohol use, and the sample that produces the lowest probability of alcohol use.
In [44]:
predclass,prob,_ = nnet.use(X,allOutputs=True)
In [45]:
prob.shape
Out[45]:
In [46]:
np.argmax(prob[:,1])
Out[46]:
In [47]:
prob[100,:]
Out[47]:
In [48]:
highest = X[np.argmax(prob[:,1]),:]
lowest = X[np.argmin(prob[:,1]),:]
print('Hi Alc Lo Alc')
for h,l,n in zip(highest,lowest,Xnames):
print('{}\t{}\t{}'.format(h,l,n))
Let's do this again with a larger network of 20 units.
In [55]:
hidden = 20
plt.figure(figsize=(20,10))
nnet = nn.NeuralNetworkClassifier(X.shape[1],[hidden],len(np.unique(T)))
nnet.train(X,T,nIterations = 1000)
plt.subplot(1,2,1)
plt.plot(np.exp(-nnet.getErrorTrace()))
plt.subplot(1,2,2)
nnet.draw(inputNames = Xnames)
In [56]:
predclass,prob,_ = nnet.use(X,allOutputs=True)
highest = X[np.argmax(prob[:,1]),:]
lowest = X[np.argmin(prob[:,1]),:]
print('Hi Alc Lo Alc')
for h,l,n in zip(highest,lowest,Xnames):
print('{}\t{}\t{}'.format(h,l,n))
In [ ]:
By forcing all information to flow through a very narrow layer, a small-dimensional representation of each sample can be obtained.
In [57]:
hiddens = [10,2,10]
plt.figure(figsize=(12,8))
nnet = nn.NeuralNetworkClassifier(X.shape[1],hiddens,len(np.unique(T)))
nnet.train(X,T,nIterations = 500)
plt.subplot(1,2,1)
plt.plot(np.exp(-nnet.getErrorTrace()))
plt.subplot(1,2,2)
nnet.draw(inputNames = Xnames)
In [58]:
predclass,prob,Z = nnet.use(X,allOutputs=True)
highest = X[np.argmax(prob[:,1]),:]
lowest = X[np.argmin(prob[:,1]),:]
print('Hi Alc Lo Alc')
for h,l,n in zip(highest,lowest,Xnames):
print('{}\t{}\t{}'.format(h,l,n))
In [59]:
len(Z)
Out[59]:
In [60]:
bottleneck = Z[1]
bottleneck.shape
Out[60]:
In [61]:
plt.figure(figsize=(10,10))
plt.scatter(bottleneck[:,0],bottleneck[:,1],c=T, s=60, alpha=0.3);
In [62]:
hiddens = [10,10,2,10,10]
plt.figure(figsize=(12,8))
nnet = nn.NeuralNetworkClassifier(X.shape[1],hiddens,len(np.unique(T)))
nnet.train(X,T,nIterations = 1000)
plt.subplot(1,2,1)
plt.plot(np.exp(-nnet.getErrorTrace()))
plt.subplot(1,2,2)
predclass,prob,Z = nnet.use(X,allOutputs=True)
bottleneck = Z[int(len(Z)/2)]
plt.scatter(bottleneck[:,0],bottleneck[:,1],c=T, s=60, alpha=0.3);
We should partition data into training and testing subsets, to see if the two-dimensional mapping learned on training data cleanly separates testing data.
In [70]:
def percCorrect(predicted,T):
return np.sum(predicted == T) / T.shape[0] * 100
In [78]:
for Xtrain,Ttrain,Xtest,Ttest in ml.partitionsKFolds(X,T,4,validation=False,shuffle=True,classification=True):
hiddens = [100,50,10,2,10,50,100]
nIterations = 10
plt.figure(figsize=(15,8))
nnet = nn.NeuralNetworkClassifier(Xtrain.shape[1],hiddens,len(np.unique(Ttrain)))
nnet.train(Xtrain,Ttrain,nIterations = nIterations)
plt.subplot(1,3,1)
plt.plot(np.exp(-nnet.getErrorTrace()))
plt.subplot(1,3,2)
predclassTrain,prob,Z = nnet.use(Xtrain,allOutputs=True)
bottleneck = Z[int(len(Z)/2)]
plt.scatter(bottleneck[:,0],bottleneck[:,1],c=Ttrain, s=60, alpha=0.3);
plt.title('Train data')
plt.subplot(1,3,3)
predclassTest,prob,Z = nnet.use(Xtest,allOutputs=True)
bottleneck = Z[int(len(Z)/2)]
plt.scatter(bottleneck[:,0],bottleneck[:,1],c=Ttest, s=60, alpha=0.3)
plt.title('Test data')
print('Hiddens',hiddens,'nIterations',nIterations)
print('Train % correct',percCorrect(predclassTrain,Ttrain))
print('Test % correct ',percCorrect(predclassTest,Ttest))
break
In [ ]: